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; Abstract 

2 . We investigated how the stabihty of macroscopic states in the associative mem- 

^ I ory model is affected by synaptic depression. To this model, we applied the 

C/2 , dynamical mean-field theory, which has recently been developed in stochastic 

neural network models with synaptic depression. By introducing a sublattice 
method, we derived macroscopic equations for firing state variables and depres- 
2 i sion variables. By using the macroscopic equations, we obtained the phase dia- 

gram when the strength of synaptic depression and the correlation level among 
__i ^ stored patterns were changed. We found that there is an unstable region in 

pH . which both the memory state and mixed state cannot be stable and that vari- 

Q ' ous switching phenomena can occur in this region. \j\ 
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1 Introduction 

An associative memory model is one of typical neural network models that has 
discretely distributed fixed-point attractors as stored patterns. [TJ [21 [3J 21 [S] 
For this model, it is known that the macroscopic state of the network usually re- 
mains in an attractor. On the other hand, synaptic plasticity can destabilize the 
network and induce a periodic or aperiodic itinerancy of the macroscopic state. 
[BJ [71 ISl [S] Such switching phenomena, known as the dynamics among quasi- 
attractors, [lOIITTlfT^ITB] have rich implications for the field of neurodynamics, 
e.g., communication from external world, memory search or symbol-emergence. 
The switching phenomena can be observed by a model with synaptic depression, 
[6l \8\ which is a physiological phenomenon in which high-frequency presynaptic 
inputs induce a decrease in synaptic weights. [TH [121 [T^ 

In this study, we considered the associative memory model with synaptic de- 
pression. We considered correlated memory patterns as well as uncorrelated 
patterns as a general case. For this, we needed to treat not only the memory 
state but also the mixed state, which are both attractors. The mixed state is 
the mixing of arbitrary memory patterns generated by correlation learning and 
is not simply a side effect that is unnecessary for information processing. [T71 [TH] 
Recently, Igarashi et al. have proposed a dynamical mean-field theory of models 
with stochastic neurons [12]. We applied this theory to the associative memory 
model and introduced the notion of a sublattice. By using the dynamical mean- 
field theory with sublattice method, we investigated how the strength of synaptic 
depression and the correlation level among stored patterns affect the stability 
of the memory and mixed states. We found an unstable region in which both 
the memory and mixed states cannot be stable. In this region, there are various 
switching phenomena among the attractors. 

The rest of this paper is organized as follows. Section 2 describes the model 
used in this paper. In §3, we discuss the theoretical method for the model in 
order to derive the macroscopic equations by the mean-field approach and the 
sublattice method, and we also introduce a stability analysis of the steady state. 
Section 4 presents results for how synaptic depression influences the stability 
and dynamics of the macroscopic state according to the correlation level among 
stored patterns. In §5, we summarize the results presented in this paper. 

2 Model 

We discuss an attractor network model with N fully connected binary neurons. 
If the i-th neuron fires at time t, its state is Si{t) = 1: otherwise, Si(t) = 0. 
Then, the state of the network is characterized by s{t) = (si(t),--- ,SN{t)). 
The synaptic weight Jij (t) from presynaptic neuron j to postsynaptic neuron i 
at time t changes dynamically owing to synaptic depression. We use synaptic 
weight Jij (t) incorporating the synaptic depression by a fixed synaptic weight 
Jij multiplied by a dynamic amplitude factor Xj{t): 

J,,{t) = J,,x,{t), (1) 

1 — X (t) 

x,it+l) = x,{t) + -^^~-Ux,{t)s,{t), (2) 



where Xj{t) is determined by a phenomenological model of synapses, [13 HH H] 
and takes < Xj{t) < 1, where Xj{t) = 1 correspond to the case without synap- 
tic depression. The depression variable from the prcsyanptic neuron, Xj{t + 1), 
decreases by a certain fraction Uxj{t) after each spike is emitted, Sj{t) — 1, and 
recovers with time constant r. 

The system is simultaneously updated, i.e, the synchronous rule, and each neu- 
ron obeys probabilistic dynamics: 

Prob[s,(t+l) = l]=.9^(/i,(t)), (3) 

where /3 = 1/T represents the inverse temperature. The function hi{t) is the 
internal potential of the i-th neuron at time t, which is defined using the synaptic 
weight and the i-th neuron's state Si{t) as 

/i.W=E^»^(2s,(i)a;,(t)-l). (4) 

In the case without synaptic depression, the system in eq. ^ returns to the 
well known Ising spin system with <T(i) ~ (ai(t), ■ ■ ■ ,aN(t)), 

We consider the associative memory model with correlated memory patterns. 
We introduce a parent pattern and p child patterns. 

i = (Ci,---,^Ar), r-(er,---,c^), fi = i,---,P- (6) 

These are random variables drawn from the following probability distributions: 

Prob[e. = ±1] = ^> Prober = ±1] = ^^, (7) 

where correlation coefficient b takes < 6 < 1 and represents the correlation 
level between stored patterns. For & = 0, child patterns are mutually orthogonal 
for N ^ oo; for 6=1, they are the same as the parent pattern. In this study, 
we treated child patterns as memory patterns, i.e., stored patterns, so at the 
thermodynamic limit of iV — > oo, the direction cosine between memory patterns 
can be described as 



1 ^ 
l^E^f^f = S,,,+b\l^S,,,), (8) 



where 6^i, is Kronecker's delta defined as 

"^^^ " \ (p^ v). ^^> 

The distance relationships among three stored patterns are shown in Fig. 1(a). 
In this example, the mixed state is defined as follows 
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(b) 
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Mixed state 



Figure 1: (a) Schematic illustration of correlated memory patterns for p — 5. 
The direction cosines are cos^o = b and cos 9 — b^ for eq. ([5]). (b) Relations 
among the memory patterns and the mixed state. 



where the output function sgn(-) is 



sgn(u) = 



1 (u > 0) 

-1 (u<0). 



(11) 



Figure[TJb) shows a schematic diagram containing the mixed state, which corre- 
sponds to Fig. 1(a) viewed from above. The center dot in the triangle represents 
the mixed state. 
The fixed synaptic weight J^- is set, according to the Hebbian rule, to 



p 



^. = ^E^re 



fj- 



(12) 



A self-connection J„ is assumed not to exist. 



3 Analysis 

Since the synaptic weight Jij{t) = JijXj{t) is asymmetric due to the dynamic 
amplitude factor Xjit), we cannot treat the present system described by eqs. 
(HJ-© by the conventional equilibrium statistical mechanical approach. In 
this section, we analyze the associative memory model with correlated mem- 
ory patterns by a dynamical mean- field theory for finite temperature [TQ:. Then, 
we derive the macroscopic steady-state equation by introducing the sublattice 
method. 



3.1 Mean- field analysis 

First, we consider the thermal average of the i-th neuron at time i, {si{t)) = 
{gp{hi{t))). Then, we get the following equations from eqs. ^ and (H]) by using 
the mean-field approximation. 



{s,{t)) = gp{{h,{t))), {h,{t)) 



^Jy(2(s,(t)x,(t))-1), 



(13) 



where (•) denotes the thermal average with respect to eq. ([3]). Similarly, we 
take the thermal average of eq. © for the dynamic amplitude factor, 

{x,{t + 1)) = {x,{t)) + ^ ~ ^^'^^^^ - U{s,{t)x,{t)). (14) 

T 

If the number of memory patterns p is on the order of 1 with respect to the 
number of neurons N, the fixed synaptic weight Jij described by eq. P^ is 
on the order of 1/N. In this case, the equal time correlation between Si{t) and 
Xi{t) is on the order 1/A'^. At the thermodynamic limit N — >■ oo, we can consider 
Si{t) and Xi{t) as being independent. The thermal average of the product of 
Si{t) and Xi{t) can therefore be decoupled as [20] 

{.s,it)x,it))^{s,it)){x,it)). (15) 

By using this, we can rewrite eqs. ([T^ and (fH)) . respectively, as 

m,{t + l) = 5J^J,,(2m,(t)X,(t)-l) , (16) 

X,it + 1) = X,(t) + i— ^-[/m,(t)X,(t), (17) 

T 

where mi{t) = {si{t)) and Xi{t) = {xi{t)). Considering the steady states for 
the thermal average, rrii = mi{oo) and Xi = Xi(oo), respectively, we obtain the 
following microscopic equation from eqs. (|16l) and (IT71) . 

v-^ / 2m-j 
\ J J 



\J^^ V 1 + 7"ij / / 1 + im, 

where 7 = tU indicates the level of synaptic depression in the steady state. For 
the above equations, the thermal average of the firing rate and the dynamic am- 
plitude factor are determined by one depressing parameter 7 in the steady state. 
Note that in previous studies, the correlation between the dynamic amplitude 
factor and the neuron's state was taken to be 

\ 1 + 7Sj / 1+7 

since Si takes a binary value, Si — {0,1}, at a low temperature T ^ (1 <C 
/? < c»).[3T] [32] By contrast, we extend the correlation to the case of finite 
temperature in the form 

(siXi) = rriiXi = ——^ — . (20) 

1 + 7TOi 

3.2 Macroscopic steady-state equation 

In this section, we obtain the macroscopic steady-state equations from the mi- 
croscopic ones described by eq. ([T5]) in terms of the sublattice notion. This 
method is essential to describe the macroscopic state of the network [23]. By 



substituting the Hebbian rule of eq. ([T^ into eq. ([T5| , we derive the following 
microscopic equation. 



To obtain the macroscopic steady-state equation, we define p dimensional mem- 
ory patterns ^j = {^l,- ■ ■ ,^i)'^ & {~1: ^Y^ where the superscript "^ stands for 
transposition. On the basis of these vectors, a set of neurons {I,--- ,N} is 
divided into 2^ groups as 

Ir, = m.=r]}, {l,--,N}^[jlr„ (22) 

where Xr/ is called a sublattice and rj — (ry^,?]^, ..., ryP)-^ € {— 1,1}^-[21] For 
example, in the case of three memory patterns divided into eight sublattices, 
we can introduce the following combination. 

(23) 
Because the memory patterns are produced by eq.©, the number of neurons 
\Xri I in the sublattice Zj, is 

\U = l fj^^' if r, = (1,1,1)- (-1,-1,-1)- (24) 

[ ^^ N, otherwise, 

where b± = ^ and |I^| is 0{N) since 2^ is 0(1). 

Following the expression of the sublattice, we can rewrite the fixed synaptic 

weight (fT2)) as 




Jij 



1-J2V''V" i^elr^jelr,') (25) 

11=1 

= j^VV'- (26) 

We can regard the model in this study as an extended Hushimi-Temperly (HT) 
model because the synaptic weight between neurons within the same sublattice is 
constant, i.e., Jij — -^. Since the synaptic weight between neurons is determined 
by which sublattice they belong to (see eq. ([26]) ). the firing rate, the internal 
potential, and dynamic amplitude factor of neurons within the same sublattice 
are the same. Therefore, we can introduce the sublattice firing rate m^, internal 
potential /i^, and dynamic amplitude factor X^ as ?7i^ = rrii = rrij, h^j = hi — 
hj , and X^ = Xi = Xj on the condition that i d X^i and j £ Tri- 
By eqs. ([T8)) and ([26)1 . the macroscopic steady-state equations are given by: 

rrir, = gpiihr,)), Xr, = — , (27) 

1 +7"!,, 



where p,, = ^-^ denotes the relative sublattice size. 

Next, we introduce the closeness between the state of system s(t) at time t and 

the yU-th memory pattern £^^ characterized by an overlap 

1 ^ 
M^{t) = -Y.i^{2s,(t)-\). (29) 

i=\ 

By following sublattice method, we can also describe the above equation in the 
steady state as 

AfA' = ^p^,^'^(2m^, -1). (30) 

rj' 

If state s(t) corresponds to memory pattern ^'', then M^(t) is exactly 1 at 
iV — > oo. The purpose of the sublattice method is to treat the macroscopic vari- 
ables, 2^ sublattices, instead of microscopic variables, TV neurons, by grouping 
homogeneous neurons with respect to memory patterns ^^ = {Q^ ...,^['). 

3.3 Stability analysis of macroscopic steady state 

In this section, we discuss the stability of eqs. (|16l) and (IT71) . If the neurons 
belong to the same sublattice, we consider the steady state of the neurons to be 
the same state. First, in order to make a correspondence between the neuron 
index and sublattice index, we relabel the i-th neuron in the sublattice Ijj using 
index Z, 

% ^ (r,,0, / = 1,---,|I^|. (31) 

Under this mapping, the time-dependent firing rate and dynamic amplitude 
factor can be expressed by 

m,(i) '^m1{t), (32) 

X^(t)^X]'{t). (33) 

Next, we rewrite the above functions as 

m;'(i) = TO^ + 5m;'(t), (34) 

X^(t) = X^ + «,^(t), (35) 

where bvtvlit) and bX^(t) denote the small deviations around steady point tti^ 
and Xn respectively. Here, if the neurons belong to the same sublattice, we 
consider their steady states to be the same. Form eqs. (TTSl) . dTTt . and (l26l) . 
these fluctuations are 

8m^i{t + 1) = 4/3m^ (1 - '"'7) ]^ E E E '?''V^(^r,'<5™;?'(t) + 77i^,<5X,7'(i)), {I' e 1^0, (36) 

«r, + l,.-CX,*„.rw+(l-i-a„.,)«,-«,.(37, 

where we use the relation J2j=ii^ji^) "^ J2j=i^ji^) ^^^ "^y ~ 0{1/N). The 
sequence of small deviations, {Sm^} and {SXj'}, can be rewritten by Fourier 



transformation as follows: 

And the inverse Fourier transformations are given by 

k 

sxj'it) - J2sxj;{t)e^^"''/\^^\. 



(38) 
(39) 

(40) 
(41) 



By using these equations, we can rewrite the dynamics of these functions de- 
scribed by eqs. ([5S)) and ([57]) in the following form. 



IVI 



yr,"^ 



k ri' l' = l fi=l k' 

x{Xr,>d7hl{t) + m^„5lfc^;(t))e2-'='''/IVI(42) 



Y^ 5X]>[t + l)e2-i'=V|x.| ^ ^ LuX^Smlit) + (l - - - Um^ 5XZ{t)\ 

k k' ^ \ T J ) 

where i is the square root of —1. 

Since Fourier components are orthonormal, we obtain the following equations 

by comparing the 0-th coefficients in the above equations: 



2TT\k'l/\Ir,\ 



Wi P 



dm^{t+l) = 4/3m^(l - m^)l^ ^^ E ^^'?'n^^"5m^'(^) + ™^'^^?( 

rj l' = l 11=1 

= AI3mr,il~mr,)J2Pv'V-v'iXr,>6m^'{t) + mr,'6X^\t)), (45) 
v' 

(46) 



SX^{t + 1) = -UXrjSm^it) + ( 1 Unir, \ 5X^{t). 

Therefore, we get the following equation written in matrix form 



SX' 



\ sxr--' 



(t+i) 



= H 



5rhf^ 
5X 



( + !,■■■ , + 1) 



(0 
{t) 



(47) 



(t + 1) / V <5X<-^'-'-^' {t) J 



where H \s a. 2^+^ x 2^+^ matrix consisting of four 2^ x 2^ block matrices A, B, 
C, and D. 



H 



A B 
C D 



(48) 



(43) 



From eqs. (|45p and (pS]). the elements of the 2^ x 2^ block matrices are given 

by 



Br, 



4/3to^(1 - mn)pn'r] ■ v'^r,', 



^rjrf' f^ -^77^7777' 1 



£), 



TJTj' 



1 Unir, ) (5,,,,' . 



Next, by considering the fc-th coefficient in eq. P^ . we obtain: 



IV I 



M^'M 



X {Xr,, Sm^' (t) + rur,, SX^' (t) )e2""='' / l^v I 
= 4/3m^(f-m^)— ^77- 77' 



77' 



|iv 



(49) 
(50) 
(51) 

(52) 



(53) 



:iXr,'6m^'{t)+mr,,6X]]\t)) ^ e2'^"=''/IVI(54) 

i'=i 



Here, by eq. (J40L the following relation holds. 



ge2-'=''/IVI=0, (k^O). 



i'=i 



Consequently, the higher-order coefficients of eqs. (02]) and (^5)) are 
(5m^(t + l) = 0, 

6X;;it + 1) = -UXr,5rnl{t) +(l-^- Um^\ SX^{t). 
and the above equations are represented in matrix form as 



Srhi. ' ' [t 



sx, 



k 

(+1, 



(^ 



H' 






sx^ 



1)" 

(+!,■■■, + 1)^ 



it) 
it) 



(55) 

(56) 
(57) 



(58) 



From eqs. (I5B1) and (157)) . the 2^+^ x 2^+^ matrix i/' can be written as 

^' - {c d)^ (59) 

where the elements of each block matrix, C and D, are given by the same form 
of eqs. ([5T|) and (|5^. respectively. The eigenvalues of i/' are 



Ar, = 0, 1 Urrin. 

T 



(60) 



(a)r=0.0 
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Figure 2: Dependence of overlap M^ on finite temperature T at r = 100.0 and 
b = 0.2. (a) Without synaptic depression, i.e., 7 = 0.0. (b) With synaptic 
depression at 7 = 0.5. Solutions on solid lines are stable, while those on dashed 
lines are unstable. Black dots were obtained by computer simulation with N — 
9.6 X 10^. 



Since the above eigenvalues do not exceed lforr>l,0<C/<l and < m^ < 
1, the higher-order components do not influence the stability of the steady-state 
solution. Therefore, we need to investigate only the eigenvalues of the Hessian 
matrix described by eq. (|48|) in order to obtain the stability of the steady states. 
In other words, if the maximum eigenvalue |A|maa; of H satisfies the condition 
|A|rnaa: < 1, the macroscopic steady state described by eq. (P7| is stable. 



4 Results 

In this section, we present the results of our investigation of the macroscopic 
behavior of the associative memory model with synaptic depression. We consid- 
ered the associative memory model embedded with correlated memory patterns 
described by eq. (^5]) . 



4.1 Stability analysis for finite temperature 

We examined the stability of the steady state for finite temperature by numer- 
ical analysis using eq. ([27]) under various conditions. 

There are three kinds of solutions to eq. (|27p . which are called the "memory 
state" , "mixed state" , and "paramagnetic state" . The memory state corre- 
sponds to the state near a memory pattern. In this state, for example, the 
overlap is represented by {M'^,M'^,M^) = {M,M',M') with \M\ > \M'\. The 
mixed state corresponds to the state that is in the center among memory pat- 
terns, i.e., sgn(£^ -I- ^^ -I- ^^). In this state, the overlap of every memory pattern 
is the same, i.e., {M^jA/PjM^) = {M,M,M). In the paramagnetic state, each 
neuron's state is random. Thus, the overlap of every memory pattern is 0, i.e., 
(M1,M2,M3) = (0,0,0). 

Figures [2lJ a) and[2l^b) show how the overlap M^ defined by eq. (l29l) depended 
on temperature T at r = 100.0 and b = 0.2. We considered only the case of 
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Figure 3: Phase diagram for stability in parameter (T, r) space, (a) For a broad 
array of time constant r at 7 = 0.5 and h — 0.2. (b) Magnified view lower 
T-region of (a) showing where the transition of the unstable phase begins. Each 
line represents a phase boundary. 



M^ > in these figures, because the overlap was symmetric between positive 
and negative. The solutions on the solid lines are stable and those on the dashed 
lines are unstable. The black dots represent the numerical result obtained by 
computer simulation with N = 9.6 x 10"*. There is good agreement between the 
stable solution obtained from eq. (27) (solid line) and computer simulation of 
eq. (3) (black dots). This indicates that the framework of the sublattice method 
is appropriate for describing the macroscopic state. 

The value 7 was fixed at 0.0 in Fig. [UJa) and at 0.5 in Fig. [2Jb). The former 
represents the case without synaptic depression. We can divide the results in 
Fig. [5Ja) into five phases on the basis of the solutions of eq. (P7)) . In the 
memory state phase, denoted by "ME" , only the solutions of the memory state 
were stable. In the mixed state phase, denoted by "MI", only the solutions of 
the mixed state were stable. At a low temperature, there was bistable phase 
"B", in which both the memory and mixed states could be stable. At a high 
temperature, the state went into paramagnetic phase "P" . 
On the other hand, there is region "U" in Fig. HKb). We call this region an 
unstable phase. In this phase, there was no steady state. By comparing Figs. 
[IJa) and [2Kb), we see that the unstable phase arose from the effect of synaptic 
depression at intermediate temperatures. As described in the next section, in- 
teresting behavior of the macroscopic states could be observed in the unstable 
phase. 

In this section, we show phase diagrams when the depression time constant r 
and memory pattern correlation coefficient b were changed together with tem- 
perature T . In particular, we investigated in what parameter region unstable 
phase occurred. 

First, we show the phase diagram of the macroscopic state with respect to r at 
7 = 0.5 and b = 0.2 in Figs. EJa) andEKb). 

Figure EKa) shows the phase diagram at larger r, while Fig. [2Ib) shows 
it at smaller r. In these results, we can see that each transition temperature 
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Figure 4: Phase diagram for stability in paranieter (6, T) space, (a) Without 
synaptic depression, i.e., 7 = 0.0. (b) With synaptic depression at 7 = 0.5 and 
r = 100.0. Thick soHd hues represent phase boundaries of the paramagnetic 
phase. Thin and dashed hues represent the phase boundaries of the memory 
and mixed state phases, respectively. 



was constant when r was sufficiently large. Furthermore, there was no unstable 
phase at a small r. Here, note that our model is not valid for r < 1. Even 
though parameter 7 was constant, the stability of the network could change 
depending on time constant r. 

Next, we show the phase diagram for stability according to correlation coefRcient 
h in Figs. Slja) andSlJb). In Fig. IDJa), the synapses were not depressed, i.e, 
7 = 0.0, while Fig. Hl^b) is the case with synaptic depression at 7 = 0.5. The 
thick solid, thin solid, and dashed lines indicate the transition temperatures to 
the paramagnetic, memory state, and mixed state phase, respectively. First, 
we discuss the common feature of Figs. HJ^a) and IH^b). When correlation h 
was sufficiently large, only the mixed state was stable because the distances 
between memory patterns was small. On the other hand, with small 6, both the 
memory and mixed states could be stable. In the high temperature region, the 
phase became paramagnetic. As shown in Fig. IHJb), regardless of the correlation 
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Figure 5: Periodic oscillatory behavior in the unstable phase at 7 = 0.5. The 
top figure shows the dependence of each overlap M^, M^ and M^ on time t and 
the bottom eight figures show the dependences of sublattice firing rate nij^ and 
dynamic amplitude factor X^ on time t at (a) h — 0.05, T — 0.65, (b) h — 0.8, 
T = 1.4 and (c) h = 0.35, T = 0.5. 



coefficient, the unstable phase can exist at intermediate temperatures as a result 
of synaptic depression. 

In Fig. |4j 6c means the maximum value of correlation coefficient h for which the 
memory state can be stable. We have analytically proved (see the appendix for 
details) that h^ has the same value, l/-\/2, regardless of depression parameter 7. 
Therefore, synaptic depression does not influence the stability of the memory 
state in p ^ 0(1) and /? — > 00. 

4.2 Macroscopic behavior in unstable phase 

In the previous section, we described our investigation of the macroscopic steady 
state and found the unstable phase for various parameter values. Previous 
studies have reported that the network can be unstable owing to synaptic 
depression [B]. However, the dynamics of the macroscopic state in that phase 
were not demonstrated by a dynamical equation obtained by mean-field analy- 
sis. In the present study, we investigated how the behavior of the macroscopic 
state in the unstable phase by theory and computer simulation. 
First, we assumed that neurons within the same sublattice I,, follow the same 
dynamics. Therefore, the firing rate and the dynamic amplitude factor can be 
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Figure 6: Dynamical process of a macroscopic state corresponding to Fig. [Sjc) 
in phase space composed of overlaps {M^,M^,AI^). (a) Theoretical results, (b) 
Simulation results. In this space, memory patterns, ^^, ^^, and £_^ correspond to 
(1, 6^, 6^), (&^, 1, 6^), and (6^, 6^, 1), respectively. The center dots in the triangles 
represent the mixed state. The solid lines in the triangles indicate the trajectory 
of the network state. 



described as 



mi{t) ^ rrinit), Xi{t) = X^(i), i £ Ir,- 



(61) 



Under this assumption, from eqs. (J16p and (jl7|) . we obtained macroscopic dy- 
namical equations for the sublattice mode: 



mr,{t + l) = gp{{hrj{t))), 

Xr^{t + 1) = Xrj{t) + ^^^'^^^> _ Umr,{t)Xrj{t). 



(62) 
(63) 

(64) 



For simplicity, the initial state was set to the first memory pattern ^ in the 
case without synaptic depression as described below. 



Mi(0) = 1.0, i\/2(0) = M^{Q) = 6^, 

''^^^ " I 0, [Q = -1), 
X, = 1.0. 



(65) 
(66) 
(67) 
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These equations correspond to the fonowing condition in terms of sublattice 
mode. 

-''(«) = { Ja I:;^ : %, (^«) 

Xr,{0) = 1.0. (69) 

Figures [Slja)-(c) show the dependence of the overlaps M'^{t) [jjl = 1, 2, 3) defined 
by eq. (|29l) on time t in the unstable phase. They show that different oscillatory 
behavior occurred depending on parameters h and T. Here, time t corresponds 
to the Monte Carlo step of the computer simulation. The solid lines represent 
the dependence of M^ on time t obtained by numerically solving eqs. (I62|) - (l64l) . 
The dotted and dashed lines similarly represent those of AP and M^ , respec- 
tively. The black dots in the top panels indicate M^{t) obtained by computer 
simulation with N = 9.6 x 10*. The eight bottom panels in each column show 
the dependences of rn,, and Xr^ on time t obtained by a theoretical approach. 
The thick and thin lines in these figures correspond to m,, and X,,, respectively. 
Figures [SJa) andjSJb) show good agreement between the theory represented by 
lines and computer simulation represented by black dots. However, we can see 
that the theoretical and simulation results are out of phase in Fig. [SJc). This is 
because the assumption that neurons within same sublattice I,j follow the same 
dynamics is not strictly valid. However, the periods of the dynamics derived 
from theory and computer simulation do coincide. 

The overlap M^ periodically oscillated as shown in Fig. [SJa) when correlation 
coefficient h was very small, i.e., b ~ 0.05 and T = 0.65. In this case, M^ and 
M^ also slightly oscillated keeping the same values as each other. That is to 
say, the network switched between the first memory pattern ^ and the anti 
memory pattern — £ in cycles. We see in the bottom figures that tti^ and X^ 
oscillated with the same phase when the first components of sublattice indices 
•q^ had the same value. 

When the distances between stored patterns was sufficiently small, which cor- 
responds to the case of large &, each overlap oscillated with the same value 
at & = 0.8 and T = 1.4, as shown in Fig. [5jb). In this case, the network 
periodically oscillated between the mixed and anti-mixed states. The bottom 
figures in Fig. [SJb), illustrate that rrir, and Xr, oscillated with the same phase 
when r]^ + rf' + rf had the same value. This is attributed to the mixed state, 
sgn(r7^ +1^ + rf). 

At fe = 0.35 and T — 0.5 (Fig. [5l^c)), unlike both the macroscopic behavior 
shown in Figs. [5lja) and[5l[b), the overlaps oscillated while keeping a positive 
value, i.e., M^^ > 0, and switched to each other. In other words, the network 
toured all the memory patterns in turn. We examined the oscillation at & = 0.35 
and T = 0.5 in more detail. Figure El illustrates the oscillation represented in 
Fig. \^c) within the phase space composed of {M^,M'^,M^) obtained from 
theory and simulation. The phase space is represented as a two-dimensional 
space whose plane surface contains all the attractors. The apexes of the tri- 
angles correspond to the coordinates of the memory patterns for eqs. ([5]) and 
(1^ . i.e., (1,&2,62), (6^,1, &^) and {b'^,b'^,l). The center dots in the triangles 
indicate the coordinates of the mixed state in phase space. The solid lines in the 
three triangles represent the locus of the state in a period, and the three figures 
at the top and bottom in Fig. [S] are in chronological order from left to right. 
We can see that the macroscopic state starting from the first memory pattern 
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^ did not stay in the attracter. Consequently, in such a cychc behavior, the 
state gravitated toward the mixed state once, but it could not remain there and 
immediately transited to another memory pattern. 

5 Conclusion 

In this paper, we discussed the associative memory model with synaptic depres- 
sion and applied the dynamical mean- field theory with the notion of a sublattice 
to the model by a statistical mechanical approach. We then considered a model 
that stores three correlated memory patterns, and examined how the stability of 
each steady state can change depending on the strength of synaptic depression 
and the correlation level among the memory patterns. Our theory enables us 
to treat the stability of not only the memory state but also the mixed state. As 
a result, we found that there is an unstable phase in which the network could 
not remain in any attractors. Furthermore, we investigated the macroscopic dy- 
namics in the unstable phase and showed that three different types of oscillation 
existed in that phase depending on certain parameters: the fraction of synaptic 
depression, recovery time constant, and correlation level among the memory 
patterns. The first one is the oscillation between the memory and anti-memory 
states: this oscillation has been reported in previous work[|i . The second is the 
oscillation between the mixed and anti-mixed states, which occurred owing to 
the correlation among memory patterns. The third is the oscillation in which the 
network toured among the memory patterns periodically via the mixed state. 
This switching phenomenon may be connected with a search among similar 
memories and "attractor ruins" in terms of chaos. [7]|11) 

In this study, we focused on a small number of stored patterns. In future work, 
we compute the starage capacity of the associative memory model with synaptic 
depression where we consider the case in which the number of stored patterns 
is on the order of N. 

A Stability of memory pattern at the low tem- 
perature limit 

At the low temperature limit {(3 — >■ oo), the stability of the memory state has 
nothing to do with synaptic depression as shown by FigHl We analytically prove 
this statement in this section. 
We rewrite the overlap (1^^ in the steady state as 



1 ^ 
^' - ]7E^r(2s,~l), (70) 

i 
^ T7E^^g^('^')(/'-^^)' (71) 



where we use eq. ([3]). By considering the dynamic amplitude factor in the 
steady state (xj = 1/1 + 7Sj), we can denote the internal potential hi of the 
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i-th neuron as 

^^ = iEE^"?"(2^.-^.--l)' (72) 

N p 



- ^EE^"^"(2^.-^.--l)' (73) 

3 = 1 ^"=1 
. N p , . 



i=i ^=1 



by using eqs. (Hj) and (J12p for iV — > oo and p ^ (1). Here, we use the following 
reasonable approximation at /3 — )■ oo [21 . 

(75) 



1 + 7Sj 1 + 7 
Then, eq. (j74p is rewritten as 

1 p ^ / o \ 

^^ = ^EerEc.^(T^-i)' ^''^ 

nT^E^rE^n^^.-i) 



^(i+^).ti 



= ^Al^'M-. (78) 

where we use j^ XJi=i "^j* — (A^ — >■ oo) for eq. ^ in the last step. Therefore, 
for < 7 and eq. (|7T|) . the overlap M^ is written as 

M'^ = ((sgn(f]erA/''))), (79) 

p=i 

where ((•)) represents the average with respect to stochastic variable ^^ . This 
is equal to the overlap in the case without synaptic depression at the low tem- 
perature limit (/3 — ?► oo). When the state is the first memory pattern ^ , i.e., 
[M^ .M"^ ,M'^) = (1,6^,6^), for example, eq. dZH) becomes 

l = ((sgn(l + fe2^0)- (80) 

Consequently, in the condition h < l/\/2 — be, the memory pattern phase "ME" 
is stable regardless of synaptic depression. 
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